{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wFPyjGqMQ82Q"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "aNZ7aEDyQIYU"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uMOmzhPEQh7b"
      },
      "source": [
        "# 归一化\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td><a target=\"_blank\" href=\"https://tensorflow.google.cn/addons/tutorials/layers_normalizations\"><img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\">View on TensorFlow.org</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/zh-cn/addons/tutorials/layers_normalizations.ipynb\"><img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\">Run in Google Colab</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/addons/tutorials/layers_normalizations.ipynb\"><img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\">View source on GitHub</a></td>\n",
        "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/zh-cn/addons/tutorials/layers_normalizations.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\">Download notebook</a></td>\n",
        "</table>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cthm5dovQMJl"
      },
      "source": [
        "## 概述\n",
        "\n",
        "此笔记本将简要介绍 TensorFlow 的[归一化层](https://github.com/tensorflow/addons/blob/master/tensorflow_addons/layers/normalizations.py)。当前支持的层包括：\n",
        "\n",
        "- **组归一化**（TensorFlow Addons）\n",
        "- **实例归一化**（TensorFlow Addons）\n",
        "- **层归一化**（TensorFlow Core）\n",
        "\n",
        "这些层背后的基本理念是对激活层的输出进行归一化，以提升训练过程中的收敛。与[批次归一化](https://keras.io/layers/normalization/)相反，这些归一化不适用于批次，而是用于归一化单个样本的激活，这样可使它们同样适用于循环神经网络。\n",
        "\n",
        "通常，通过计算输入张量中子组的均值和标准差来执行归一化。此外，也可以对此应用比例因子和修正因子。\n",
        "\n",
        "$y_{i} = \\frac{\\gamma ( x_{i} - \\mu )}{\\sigma }+ \\beta$\n",
        "\n",
        "$ y$：输出\n",
        "\n",
        "$x$：输入\n",
        "\n",
        "$\\gamma$：比例因子\n",
        "\n",
        "$\\mu$：均值\n",
        "\n",
        "$\\sigma$：标准差\n",
        "\n",
        "$\\beta$：修正因子\n",
        "\n",
        "下面的图像演示了这些技术之间的区别。每个子图显示一个输入张量，其中 N 为批次轴，C 为通道轴，(H, W) 为空间轴（例如图片的高度和宽度）。蓝色像素由相同的均值和方差归一化，均值和方差通过聚合这些像素的值得出。\n",
        "\n",
        "![](https://github.com/shaohua0116/Group-Normalization-Tensorflow/raw/master/figure/gn.png)\n",
        "\n",
        "来源：(https://arxiv.org/pdf/1803.08494.pdf)\n",
        "\n",
        "权重 γ 和 β 可以在所有归一化层中训练，以补偿表征能力的可能损失。您可以通过将 `center` 或 `scale` 标记设置为 `True` 来激活这些因子。当然，您也可以在训练过程中对 `beta` 和 `gamma` 使用 `initializers`、`constraints` 和 `regularizer` 来调整这些值。 "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I2XlcXf5WBHb"
      },
      "source": [
        "## 设置"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTlbneoEUKrD"
      },
      "source": [
        "### 安装 Tensorflow 2.0 和 Tensorflow-Addons"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_ZQGY_ALnirQ"
      },
      "outputs": [],
      "source": [
        "!pip install -U tensorflow-addons"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7aGgPZG_WBHg"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_addons as tfa"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u82Gz_gOUPDZ"
      },
      "source": [
        "### 准备数据集"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3wso9oidUZZQ"
      },
      "outputs": [],
      "source": [
        "mnist = tf.keras.datasets.mnist\n",
        "\n",
        "(x_train, y_train),(x_test, y_test) = mnist.load_data()\n",
        "x_train, x_test = x_train / 255.0, x_test / 255.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UTQH56j89POZ"
      },
      "source": [
        "## 组归一化教程\n",
        "\n",
        "### 简介\n",
        "\n",
        "组归一化 (GN) 将输入的通道分成较小的子组，并根据其均值和方差归一化这些值。由于 GN 只对单一样本起作用，因此该技术与批次大小无关。\n",
        "\n",
        "在图像分类任务中，GN 的实验得分与批次归一化十分接近。如果您的整体 批次大小很小，则使用 GN 而不是批次归一化可能更为有利，因为较小的批次大小会导致批次归一化的性能不佳。\n",
        "\n",
        "###下面的示例在 Conv2D 层之后将 10 个通道按标准的“最后一个通道”设置分为 5 个子组："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aIGjLwYWAm0v"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.models.Sequential([\n",
        "  # Reshape into \"channels last\" setup.\n",
        "  tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),\n",
        "  tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format=\"channels_last\"),\n",
        "  # Groupnorm Layer\n",
        "  tfa.layers.GroupNormalization(groups=5, axis=3),\n",
        "  tf.keras.layers.Flatten(),\n",
        "  tf.keras.layers.Dense(128, activation='relu'),\n",
        "  tf.keras.layers.Dropout(0.2),\n",
        "  tf.keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "model.fit(x_test, y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QMwUfJUib3ka"
      },
      "source": [
        "## 实例归一化教程\n",
        "\n",
        "### 简介\n",
        "\n",
        "实例归一化是组归一化的特例，其中组大小与通道大小（或轴大小）相同。\n",
        "\n",
        "实验结果表明，当替换批次归一化时，实例归一化在样式迁移方面表现良好。最近，实例归一化也已被用来代替 GAN 中的批次归一化。\n",
        "\n",
        "### 示例\n",
        "\n",
        "在 Conv2D 层之后应用 InstanceNormalization 并使用统一的初始化比例和偏移因子。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6sLVv-C8f6Kf"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.models.Sequential([\n",
        "  # Reshape into \"channels last\" setup.\n",
        "  tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),\n",
        "  tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format=\"channels_last\"),\n",
        "  # LayerNorm Layer\n",
        "  tfa.layers.InstanceNormalization(axis=3, \n",
        "                                   center=True, \n",
        "                                   scale=True,\n",
        "                                   beta_initializer=\"random_uniform\",\n",
        "                                   gamma_initializer=\"random_uniform\"),\n",
        "  tf.keras.layers.Flatten(),\n",
        "  tf.keras.layers.Dense(128, activation='relu'),\n",
        "  tf.keras.layers.Dropout(0.2),\n",
        "  tf.keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "model.fit(x_test, y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qYdnEocRUCll"
      },
      "source": [
        "## 层归一化教程\n",
        "\n",
        "### 简介\n",
        "\n",
        "层归一化是组归一化的特例，其中组大小为 1。均值和标准差根据单个样本的所有激活计算得出。\n",
        "\n",
        "实验结果表明，层归一化非常适合循环神经网络，因为它可以独立于批大小工作。\n",
        "\n",
        "### 示例\n",
        "\n",
        "在 Conv2D 层之后应用 Layernormalization 并使用比例和偏移因子。 "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Fh-Pp_e5UB54"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.models.Sequential([\n",
        "  # Reshape into \"channels last\" setup.\n",
        "  tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),\n",
        "  tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format=\"channels_last\"),\n",
        "  # LayerNorm Layer\n",
        "  tf.keras.layers.LayerNormalization(axis=1 , center=True , scale=True),\n",
        "  tf.keras.layers.Flatten(),\n",
        "  tf.keras.layers.Dense(128, activation='relu'),\n",
        "  tf.keras.layers.Dropout(0.2),\n",
        "  tf.keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "model.fit(x_test, y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "shvGfnB0WpQQ"
      },
      "source": [
        "## 文献\n",
        "\n",
        "[Layer norm](https://arxiv.org/pdf/1607.06450.pdf)\n",
        "\n",
        "[Instance norm](https://arxiv.org/pdf/1607.08022.pdf)\n",
        "\n",
        "[Group Norm](https://arxiv.org/pdf/1803.08494.pdf)\n",
        "\n",
        "[Complete Normalizations Overview](http://mlexplained.com/2018/11/30/an-overview-of-normalization-methods-in-deep-learning/)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "layers_normalizations.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
